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1 INTRODUCTION 

This special case study report presents the Science and Engineering Technical Assessments 
(SETA) team’s findings for exploring the correlation between the underlying models of 
Advanced Risk Reduction Tool (ARRT) relative to how it identifies, estimates, and integrates 
Independent Verification & Validation (IV&V) activities. The special case study was conducted 
under the provisions of SETA Contract Task Order (CTO) 15, of NASA contract NAS2-98028, 
and the approved technical approach documented in the CTO-15 Modification # 1 Task Project 
Plan. 


Section 2 provides an ARRT project overview and executive summary of SETA’s primary 
finding. Section 3 describes the analyses performed for this Special Case Study; provides 
examples of potential ARRT risk-methodology enhancements; and makes recommendations for 
the direction of future tool development. Section 4 compares the approach pursued by the Jet 
Propulsion Laboratory (JPL) research team to that pursued by Department of Defense (DoD) and 
the nuclear industry to reduce software acquisition and operational risks. SETA’s findings and 
recommendations are summarized in Section 5. 


2 ARRT PROJECT OVERVIEW 

ARRT is a comprehensive software tool being developed by the JPL in collaboration with the 
Glenn Research Center, West Virginia University, Texas A&M University, and Miami 
University. When fully realized, ARRT is to provide a standard means for: 

• Identifying software risks; 

• Creating optimal risk-mitigation plans; 

• Producing consistent cost and schedule risk-reduction budget estimates; and 

• Creating equitably negotiated IV&V, software development, and Quality Assurance 
(QA) plans that are compliant with NASA policy and ISO 9000 process criteria. 

Note that the last requirement is actually a compound of three requirements, in that the tool must 
be able to create three distinct types of plans: IV&V, software development, and QA. 

2.1 Executive Summary 

The evaluated version of ARRT is only capable of supporting software development-lifecycle 
planning; it does not support IV&V or QA planning. The balance of this report discusses why 
this is so, and provides recommendations for ways to expand ARRT capability into the domain 
of IV&V planning, which is the primary interest of the NASA Software IV&V Facility. 


3 SPECIAL CASE STUDY EVALUATION 

The Statement of Work for this Special Case Study prescribed four distinct analysis activities: 
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1 . [Evaluate the] IV&V activities and how they correlate to Capability Maturity Model 
(CMM) Key Practice Areas (KPA) and Software Engineering Institute (SEI) Risk 
Element (RE). 

2. [Determine what] risk mitigation activities are missing or are inappropriate. 

3. [Evaluate resource] estimation efficiency. 

4. [Analyze the] relationships between risk mitigation activities and the risks. 

The third task was deleted from the Special Case Study when it was learned that AskPete, the 
cost-modeling database, is still being developed at the Glenn Research Center. Expectations for 
each of the remaining activities were clearly defined on 18 September 2000, via a teleconference 
attended by the SETA team, and Mr. Marcus Fisher and Mr. Ken McGill of the NASA Software 
IV&V Facility. The approved technical approach was captured in the CTO-75 Modification #7 
Task Project Plan. The following sections define the revised tasks, present findings, and offer 
recommendations for the direction of future tool development. 


3.1 ARRT Failure Mode (FM) Analysis 

The goal of this analysis task was to determine the level of correlation between the Failure 
Modes (FM) defined in ARRT and SEI RE. 

3.1.1 Analysis Approach and Findings 

The results of the analysis revealed that the ARRT Failure Modes map directly to the Risk 
Elements defined by the SEI; in fact, we have since learned that the FMs were copied verbatim 
from the SEI Technical Report “Taxonomy-Based Risk Identification,” CMU/SEI-93-TR-6, 
ESC-TR-93-183. 

3.1.2 Recommendations 

Since 100% correlation exists between the two lists, no further analysis is required or 
recommended. Appendix A presents the SEI Risk Element Taxonomy versus the ARRT Failure 
Modes. 


3.2 ARRT Preventive measures Analyses Control Tests (PACTs) 
Analysis 

The goal of this analysis task was to determine the degree to which the PACTs 1 defined in ARRT 
represent the kinds of risk-mitigation activities performed by IV&V contractors. 


1 PACT is an acronym for: Preventative measures (e.g. design rules, material and parts selection, architecture, 
redundancy), Analyses (e.g. structural, optical, chemical, electrical performance, FMECAs and other reliability 
analyses), process Controls (e.g. inspections, coupon sampling, standard procedures and processes) Tests (e.g. 
functional, environmental, stress screening) 
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3.2.1 Analysis Approach and Findings 

The SETA team first inspected the PACT detailed listing and hierarchy trees to (1) determine 
whether the activities described sufficiently represent the full scope of activities that are typically 
conducted by IV&V practitioners, and (2) identify missing or inappropriate activities. 

Our inspection revealed that the PACTs are software development-best practices derived from 
SEI Key Practice Areas (KPAs) - they are not typical IV&V activities. After discussing this 
finding with the NASA customer, it was decided that a more useful approach would be to 
develop examples of how the ARRT Failure Modes might be mapped to typical IV & V risk- 
mitigation activities. The result is provided in Appendix B and represents SAIC’s best judgment 
based upon 15 years of experience in conducting IV&V for various projects. 

3.2.2 Recommendations 

A tradeoff between the cost of implementing a best practice (PACT) and its effectivity exists. 
Thus, while IV&V can always add unique value to a program due to its financial and managerial 
independence from the developer, the composition and required stringency of IV&V support is a 
function of the unique characteristics of the program and the development environment. To 
achieve the right balance requires that these factors be considered in a systematic way when 
tailoring an IV&V activity lifecycle. Figure 3-1 illustrates this concept as applied to ARRT (as 
well as a graphical depiction of the focus areas of this Special Case Study). 


Figure 3-1 Balancing Best Practices, Risks, and IV&V 
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Because the current PACTs are derived from SEI KPAs, they are by definition good software 
development practices, rather than complementary or reinforcing IV&V activities. The “see- 
saw” at the bottom represents how a balance between good software development practices and 
supporting IV&V is required to efficiently minimize risk. 


Thus, while the visual depiction of risks that ARRT now provides is useful for manually 
balancing costs versus specialized development and IV&V activities, the tool should also include 
heuristics or algorithms to find the most efficacious mix of PACTs, by weighing activity costs 
provided by AskPete against validated PACT effectivity weightings. 

SETA understands that the current PACT effectivity weightings are preliminary. Once the fullest 
list of PACTs is assembled (i.e. a list that includes additional specialized software development 
and IV&V activities) the relative ranking of the activities and their effectivity values should be 
re-examined, preferably with the goal of achieving a consensus from amongst a panel of experts 
backed by empirical evidence, if available. 

In summary, SETA recommends: 

1 . Add rV&V-specific activities to the PACT list, as the current list does not include 
specific IV&V risk-mitigation activities. This will render the list compliant with the 
requirement to create IV&V plans. 

2. Develop an expert consensus for the relative rankings of the PACTs via the Delphi 
technique, and assign effectivity values developed using a decision science method such 
as the Analytical Hierarchy Process (AHP). This research will render the output of 
ARRT defensible to potential Center customers. 

3. Develop a strategy for optimally balancing IV&V and software development PACTs 
using validated PACT effectivities and AskPete cost data. Such research would add 
significant value to the ARRT tool by making its recommendations more analytical and 
repetitive. 

3.3 ARRT FM-to-PACT Mapping Analysis 

The goal of this task was to evaluate the suitability of the mapping of PACTs to FMs currently 
defined in ARRT. 

3.3.1 Analysis Approach and Findings 

SETA’s original approach was to assess each link as being justified, marginal, or unjustified in 
SETA’s judgment. The rationale for adding or deleting specific linkages would also be provided. 

After analyzing the PACTs, SETA concluded that most of the PACTS should be performed on 
any project because they represent good engineering practices. Rather than report the obvious, 
SETA decided to do the following: 
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1. Categorize the current PACT tree into two classes: Standard Good Practices and Special 
Practices. 

2. Define additional examples of Special Practices, which would reduce risk in key areas 
when the KPA-based PACTs are already in place. 


The idea is that Standard Good Practices be considered the default list when creating a Software 
Development Plan, while the Special Practices will target project-specific areas of weakness. 


Appendix C provides SETA’s classification of the current, KPA-based PACTs into Standard 
Good Practices and Special Practices. 

Appendix D presents 38 additional Special Practices conceived to complement the current list of 
generic, KPA-based PACTs, and the recommended mapping between them and the 8 1 Failure 
Modes currently in ARRT. 

3.3.2 Recommendations 

1. Divide the PACTs into Standard Best Practices (SBPs) and Special Practices 
(SPs). IV&V activities may also be grouped, with some considered SBPs and 
other, more detailed activities falling within the class of SPs. 

A. The SBPs would constitute the “default” practices NASA should expect, with 
the SPs providing domain or project specific, targeted risk reduction. 

B. PACT tailoring could then be performed to be consistent with or 
complimentary to the Capability Maturity Model - Integrated (CMMI). 
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4 OTHER PERSPECTIVES 

This section outlines the risk-management approaches pursued by industry and government 
groups who, like NASA, have an interest in reducing software acquisition and operational risks. 
Where available, SETA has listed specific risk areas and PACTs that these groups have 
identified as being critical. The JPL team may wish to further investigate these perspectives for 
inclusion in the ARRT tool as specialized PACTs. 

4. 1 Department of Defense (DoD) 

Within the DoD, no single set of Best Practices or methods for the systematic analysis of military 
system quality or reliability exists. The DoD has moved in recent times to developer-specified 
Best Practices in the name of Acquisition Reform. A DoD Program Office may scrutinize the 
processes via an SEI Audit that a potential contractor has in place before a contract is awarded, or 
may require that the contractor be certified at a certain SEI Level before a contractor is allowed 
to bid on a contract. 


Currently, a major Navy Program uses several Risk Analysis Tools. The Program Office uses the 
Technical Risk Identification and Mitigation System (TRIMS) module available free from the 
Best Manufacturing Practices, Center of Excellence (http://www.bmpcoe.org). TRIMS is a tool 
designed to help identify, quantify, and track risks in a program, and then reduce or mitigate 
these risks to acceptable levels. It works through out all phases of a program’s transition from 
initial concept to full production and life cycle support. TRIMS is based on proven risk models 
or published practices such as those of the NAVSO P-6071 Best Practices (Templates) and the 
SEI. 

One of the contractors on this program uses several tools from the Software Program Managers 
Network (http://www.spmn.com/index.html). The mission of the Software Program Managers 
Network (SPMN) is to enable managers of large-scale, software-intensive development or 
maintenance projects to more effectively manage and succeed by identifying and conveying to 
them, management Best Practices, lessons-leamed, and direct support. The tools. Risk Radar and 
Project Control Panel, are available free from the SPMN as well as excellent literature and videos 
on Best Practices in many key program areas. 

However, when Weapons Safety is involved, more formality is apparent in the DoD. A formal 
Government Weapons Safety Review Board (WSRB) is involved in the safety validation of the 
system or sub-system before any live ordinance can be used with the system. The WSRB has 
well defined processes and checklists that are used throughout the program starting with 
requirements specification through formal testing. 
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4.2 Nuclear Industry 

Four nuclear regulatory authorities worldwide (United Kingdom’s Nuclear Installations 
Inspectorate, Canada’s Atomic Energy Control Board, France's Direction de la Surete des 
Installations Nucleaires/Institut de Protection et de Surete Nucleaire, and the US Nuclear 
Regulatory Commission) have agreed on how a safety case should be handled for acceptance of 
computer-based instrumentation and control (I&C) systems in nuclear power plants. The 
following describes the good practices, i.e., PACTs, upon which they have reached consensus. 


First, the nuclear industry is only concerned with safety systems and safety-related systems. 
Power plant systems are systematically analyzed for failure modes, with an assessment conducted 
of the acceptability of the risk associated with each FM. Only systems critical to safe operation 
are analyzed. Safety systems include emergency reactor core-cooling (protection) systems and 
decay heat-removal (safety actuation) systems. Safety-related systems include radiation 
monitors, fire detection systems, etc. 

For either of these categories (safety or safety-related), a self-standing safety case is required to 
demonstrate that the: 

1. Correctness and completeness of the overall requirements specification is justified 
relative to the intended system function; 

2. System has been designed to standards compatible with its required safety integrity; 

3. Delivered system meets all aspects of its requirements specification; and 

4. Adequate means are specified to ensure performance of the system throughout its 
operational life. 2 

Early in the project the following must be considered and evaluated as part of the risk-mitigation 
plan: 

1. Safety importance; 

2. Defense-in-depth; 

3. System boundaries; 

4. Novelty; 

5. Basis of the safety case; 

6. Licensee/regulator interface; 

7. Need for Independent Assessment; 

8. Approach to be used if system licensed in another country. 

The following are the main safety case requirements that must be met during production of the 
system. The following may be considered as the nuclear-industry required PACTs: 


2 The principal standards applicable to computer-based systems are found in 0 of IEC 880 (Software for computers 
in the safety systems of nuclear power stations). 
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1. General demonstration principles for safety and safety related systems; 

2. Complete functional capability; 

3. Correct and traceable specifications; 

4. Minimizing faults in the design; 

5. Fail safe design; 

6. Operational testability; 

7. Full system testing; 

8. Well defined standards; 

9. Competent staff and team organization; 

10. Quality assurance; 

1 1 . Attention to security throughout development; 

12. Controlled change process; 

13. Well managed documents; 

14. Consistency with operating plans and procedures; 

15. Attention to human factors of operator interfaces. 

Additionally, if it is a safety system, the following must also be developed/utilized/ evidenced: 

1. Single failure criterion; 

2. Common cause failures; 

3. Structured software development process; 

4. System design principles; 

5. Complete V&V using both test and static analysis; and 

6. Use of valid and controlled tools. 

Regulators focus attention upon, or look for evidence of: 

1 . Independent assessment; 

2. Defense in depth; 

3. COTS; 

4. Formal methods; 

5. Performance feedback; and 

6. Technological developments. 
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4.3 Aviation Industry 

In the United States, the safety requirements for civil aviation systems are defined and published 
by the Requirements and Technical Concepts for Aviation (RTCA), which is an association of 
aeronautical organizations from both government and industry. RTCA is not an official agency 
of the U.S. Government, but its findings are followed as a standard guideline and are recognized 
by the Federal Aviation Administration (FAA). The findings of the RTCA are jointly developed 
with the European Organization for Civil Aviation Equipment (EUROCAE) WG-12 through a 
consensus process. RTCA Document RTCA/DO-178B, Software Considerations in Airborne 
Systems and Equipment Certification, provides guidance for airborne systems. An appendix to 
DO-178B, soon to be issued, shall provide similar guidance for ground-based systems. 


DO-178B discusses several aspects related to software development and ways to ensure that risk 
is mitigated. This is achieved through redundancy suggestions rind the use of dissimilar software 
running in parallel. It also classes software into five levels, A-E, based upon the contribution of 
software to potential failure conditions, which might provide a useful set of criteria for 
determining the level and stringency of IV&V that should be applied to a project. Also 
considered as PACTs that can mitigate risk are: 

1. System Architectural Considerations - ways in which the system safety assessment 
process can determine whether the system architecture preludes anomalous behavior. 

2. Partitioning - the technique by which isolation between functionally independent 
software components can reduce the software verification process. 

3. Multiple-Version Dissimilar Software - a design technique that involves producing 
two or more software components that provide a function in a way that can avoid 
sources of common errors. 


DO-178B also recommends PACTs associated with the software verification and quality 
assurance processes, as well as guidance for the purposes and contents of software lifecycle data 
such as software development and software verification plans. Finally, it discusses tool 
qualification, use of previously used software in safety critical applications, and alternative 
methods including the application of formal methods and exhaustive input testing, which could 
be borrowed and included as PACTs in ARRT. 

Similar standards, utilized by the European Community and Great Britain in the establishment of 
the safety case for air traffic control systems are IEC 61508, Information Technology - Software 
Process Assessment, and IEC 61508, Functional Safety of electrical/electronic/programmable 
electronic safety-related systems. 
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5 CONSOLIDATED RECOMMENDATIONS 

SETA recommends that the JPL researchers: 

1. Add IV&V-specific activities to the PACT list, as the current list does not include 
specific IV&V risk-mitigation activities. This addition will render the tool compliant 
with the requirement to create IV&V plans. 

2. Divide the PACTs into SBP and SP - IV&V activities might also be so grouped, with 
some considered SBPs and other more detailed activities falling within the class of SP. 

a. The SBPs would constitute the “default” practices NASA should expect, with 
the SPs providing domain or project-specific targeted risk reduction. 

b. PACT tailoring could then be performed along the lines of what the SEI is 
proposing for CMMI. 

3. Develop an expert consensus for the relative rankings of the PACTs via the Delphi 
technique and assign effectivity values developed using a decision science method 
such as the AHP. This research will render the output of ARRT defensible to potential 
Center customers. 

4. Develop a strategy for optimally balancing IV&V and software development PACTs 
using validated PACT effectivities and AskPete cost data. Such research would add 
significant value to the ARRT tool by making its recommendations more analytical 
and repeatable. 

5. Consider adopting selected safety and mission critical PACTs that are endorsed by 
related industries such as nuclear power, civil aviation, and DoD. 
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Appendix A ARRT Failure Modes Mapped to SEI Risk Elements 


ARRT FMs 

SEI RES 

1 

SEI SRE 

Appendix 2 of SEI CMU/SEI-93-TR-6 

2 

Product Engineering 

A. Product Engineering 

3 

Requirements Risks 

1. Requirements 

4 

Stability: Unstable requirements 

a. Stability 

5 

Completeness: Incomplete requirements 

b. Completeness 

6 

Clarity: Unclear requirements 

c. Clarity 

7 

Validity: Invalid requirements 

d. Validity 

8 

Feasibility: Infeasible requirements 

e. Feasibility 


Precedent: Unprecedented requirements f. Precedent 

Scale: Large size or high complexity system g. Scale 

Design Risks 2. Design 


Functionality: Potential problems in meeting functional a. Functionality 

specs 


13: Difficulty: Difficult design to achieve 

b. Difficulty 

14: Interfaces: ill-defined or uncontrolled internal interfaces 

c. Interfaces 

15: Performance: Stringent response time or throughput 

d. Performance 

requirements 


1 6: Testability: Product difficult to test 

e. Testability 


17: Hardware Constraints: Tight constraints because of target 
hardware 

18: Non-Developmental Software: Problems with software used 
here, but developed elsewhere 


f. Hardware Constraints 


g. Non-Developmental Software 


19 

Code and Unit Test Risks 

3. Code and Unit Test 

20 

Feasibility: Implementation of design difficult 

a. Feasibility 

21 

Unit Test: Level and time for unit test inadequate 

b. Testing 

22 

Coding/Implementation: Problems for coding (difficult 
requirements, poor design, etc.) 

c. Coding/Implementation 

23: 

Integration and Test Risks 

4. Integration and Test 

24: 

Environment: Inadequate test and integration environment 

a. Environment 


25: Product: Integration of software components to each other 
and to hardware, and testing of the product. 


26: System: Problems with integration of product to interfacing 
systems or sites 


27: Engineering Specialties Risks 


28: Maintainability: Implementation difficult to understand or 
maintain 


29: Reliability: Problems with system reliability or availability 


30: Safety: Infeasible safety requirements 


31: Security: Security requirements more stringent than state-of- 
practice or experience 


32: Human Factors: Poor human interface specification 


33: Specifications: Feasibility of implementation and quality 
attributes of stability, completeness, clarity, and verifiability. 


34: Development Environment 


b. Product 


c. System 


5. Engineering Specialties 


a. Maintainability 


b. Reliability 


c. Safety 


d. Security 


e. Human Factors 


f. Specifications 


B. Development Environment 
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ARRTFMs 

SEI REs 

35: Development Process Risks 

1 . Development Process 

36: Formality: Problems with the formality of the development 
process 

a. Formality 

37: Suitability: the adequacy with which the development model, 
process, methods, and tools support required activities. 

b. Suitability 

38: Process Control: Problems in ensuring process is used, or 
with measurement or improvement of the process. 

c. Process Control 

39: Familiarity: Project members are inexperienced in process 

d. Familiarity 

40: Product Control: Problems with traceability of requirements 
from specifications to implementation. 

e. Product Control 

41: Development System Risks 

2. Development System 

42: Capacity: Insufficient work station processing power, 
memory, etc. 

a. Capacity 

43: Suitability: Development system does not support all 
phases, activities, functions 

b. Suitability 

44: Usability: Development system is not easy to use 

c. Usability 

45: Familiarity: Unfamiliarity with development system 

d. Familiarity 

46: Unreliable development system 

e. Reliability 

47: System Support: Problems with training, access to expert 
users, or repair by vendors 

f. System Support 

48: Deliverability: Lack of resources to deliver system. 

g. Deliverability 

49: Management Process Risks 

3. Management Process 

50: Planning: inadequate planning and agreement to plan 

a. Planning 


51: Project Organization: Problems with the effectiveness of 
program organization, effectiveness of roles and 
responsibilities. 


52: Management Experience: Inexperienced managers 


53: Program Interface: Poor communication between managers 
and customer, senior management 


b. Project Organization 


c. Management Experience 


d. Program Interfaces 


54: Management Methods Risks 

4. Management Methods 

55: Monitoring: Poor project monitoring by management 

a. Monitoring 

56: Personnel Management: Problems with selection and 

b. Personnel Management 

training and work according to plan, get help, etc. 


57: Quality Assurance: Lack of quality assurance procedures 

c. Quality Assurance 

and resources 


58: Configuration Management: Poor configuration 

d. Configuration Management 

management 


59: Work Environment Risks 

5. Work Environment 

60: Quality Attitude: Lack of orientation toward quality 

a. Quality Attitude 

workmanship 


61 : Cooperation: Lack of team spirit 

b. Cooperation 

62: Communication: Poor technical communication among team 

c. Communication 

and management 


63: Morale: Low morale on project 

d. Morale 

64: Program Constraints 

C. Program Constraints 

65: Resources Risks 

1 . Resources 

66: Schedule: Inadequate or unstable schedule 

a. Schedule 

67: Staff: Staff inexperience's, lacking knowledge or skills 

b. Staff 
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ARRT FMs 


SEI REs 


68: Budget: Insufficient or unstable budget 

69: Facilities: Inadequate facilities for building and delivering 
product 

70: Contract Risks 

71 : Type of Contract: Problematic con tract 

72: Res trictions: Restrictive contract 

73: Dependencies: Program is dependent on outside products 
or services 

74: Program Interfaces Risks 

75: Customer: Problems with customer's level of skill or 
experience in the technical or application domain, not 
having access to customer factions, etc. 

76: Associate Contractors: Problems with associate contractors- 
conflicting political agendas, interface problems, lack of 
cooperation. 

77: Subcontractors: Inadequate task definitions, subcontractor 
management mechanisms, lack of knowledge of the 
program or corporation, etc. 

78: Prime Contractors: Poorly defined task definitions, complex 
reporting, or dependencies on technical or programmatic 
information 

79: Corporate Management: Poor communication and direction 
from senior management; non-optimum levels of support. 

80: Vendors: Unresponsive vendors- dependencies on 

deliveries and support for critical system components. 

81: Politics: Political problems for project 


c. Budget 

d. Facilities 


2. Contract 

a. Type of Contract 

b. Restrictions 

c. Dependencies 


3. Program Interfaces 
a. Customer 


b. Associate Contractors 


c. Subcontractors 


d. Prime Contractors 


e. Corporate Management 


f. Vendors 


g. Politics 
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Appendix C. ARRT PACTS Classified into Two Groups: Standard and 
Specialized 



Requirements ' ' : fc- ‘ ■ 


Authorization to proceed 

Identify design/coding standards 


Maintain Software Development Folder 


Software Assurance reviews Management Plan 


Implement Problem report and corrective action system 


Management Plan approval 


Documented requirements 


Peer review of requirements 


Conduct formal inspection of requirements 


Software Assurance reviews requirements 


Requirements approval 


Peer review of plans 


Implement Formal configuration management 


Conduct Product Assurance Audits 


Conduct Formal Reviews 


Document approval of requirements and formal review 


Customer approval of certification procedures 


Conduct analyses of criticality and safety 


Plan and schedule IV&V activities 


Identify method for verification of safety critical functions and requirements 


Design 

Document preliminary and detailed designs 


Peer or management reviews of design meets requirements 


Conduct peer reviews on design 

Conduct formal inspections of design products 


Conduct formal design review(s) 


Record and maintain peer and formal review results 


Document design changes 


Approval of design changes 


Baseline design 


Place design under CM control after review changes incorporated 


Implement formal change control 


Prepare verification/validation ___ 


Perform safety analyses 


Create verification procedures for safety critical functions and requirements 


development > ■ ; '• ..b ; 


Record results of peer reviews and formal reviews 


Conduct code walkthroughs, peer reviews or code inspections 


Conduct formal inspections of code 


Standard 

Good 

Practices 


Special 

Practices 
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PACTS 

41: 

Implement formal software configuration management 


42: 

Periodically perform backups 


43: 

Approval of documented design 


44: 

Document lower level test procedures 



Standard Special 

Good Practices 

Practices 


Document test plans and procedures 

Document verification/validation results 

Document and maintain test results 


Capture and document final unit tests 


Product assurance witnesses testing of safety critical and security functions 
Document design changes _____ 


Trac k change requests, problem reports and corrective action reports 

Document approval of version and build of software for release to system 
test 


Approval for product release 


System Testing 

Conduct system level integration 

Conduct testing to test procedure and record results 

Software assurance approval of all tests 

Baseline software and related documentation after passing tests 


Place test suites, simulators and test results under formal configuration 
control 


Document problems, changes and corrective actions found during 
acceptance testing __ 


Track problems, changes and corrective actions to closure 

Implement documented problem report and corrective action system for 
baselined software 


Acceptance obtained from customer 


Acceptance/Release 

Project lead verifies all requirements are met or waiver approved 

Validate project via customer witness final demonstration 


Validate product via system level test or product demonstration 


Safety Critical software approval by safety board/safety manager 


69: 

Final hazard reports reviewed and approved 


70: 

Product Assurance conducts FCA/PCA 

X 

71: 

Acceptance review prior to release 

X 

72: 

Acceptance approval documented 

X 

73: 

Written customer approval of demonstration for release authorization. 


74: Copies of all software products and documentation delivered to customer 

X 

75: 

Copy of released code and any relevant documentation, plans, reports, 
papers, test cases provided to customer 

X 

76: 

Software executables and necessary documents made available from 
configuration management system 

X 

77: 

Formal notice of release 


78: 

Identified, labeled copies of software deliverables maintained in agreed 
location 

X 


Copy of software kept by developers 
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Standard 

Good 

Practices 


Special 

Practices 




User maintains software after release 


Record design changes 


Record approved implementation changes 


Reverify and revalidate changes 


Commercial release IAW NPG 2210 


Support 

Changes documented and tracked using agreed configuration management 
process 

Changes formally controlled and approved prior to implementation 


Problem report and corrective action system continues through working life 
of product or until turned over to another organization. 


Pull, label and archive all current releases of software and documentation 
once working life of product is complete 


Independent Verification and Validation 


Perform IV&V reviews, analyses and tests 


Required Documents 


Management Plan 


Development Activities Plan 


Verification Plan ____ 


Validation Plan 


Organizational and Technical Interface Descriptions 


Requirements Documentation 


Design Documentation 


Testing Procedures __ 


Assurance Plan 


Risk Management Plan 


104 

Version Description 

X 

105 

Certification Procedures 


106 

Training Development Plan 

X 

107 

Delivery and Operational Transition Plan 


108 

Concept Documentation 


109 

Safety Assurance Procedures 


110 

Security and Privacy Procedures 


111 

Acquisition Activities Plan 


112 

Users Guide 

X 

113 

Operational Procedures Manual 
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Appendix 


D. Examples of Additional Specialized Software 

Development PACTS Mapped to Current ARRT Failure 
Modes 



Potential 
Specialized 
SWD PACTS 

PACT Description 

Mapping to ARRT Failure Modes 

SI 

Use evolutionary/incremental development model 

4 [Unstable Requirements] 

5 [Incomplete requirements] 

9 [Unprecedented requirements] 
(See 0 for additional definitions) 

S2 

Baseline stable requirements and limit work to those 

4-9, 31-32 

S3 

Prototype/simulate requirements and get user/client 
feedback 

6, 9. 13, 15, 17, 20,31,32 

S4 

Perform independent JAD or paper exercise with users or 
client to validate requirements 

7-9, 30 

S5 

Identify high-complexity items early and prototype 

10, 12, 13, 15, 17, 20 

S6 

Perform additional V&V on complex items 

10, 12, 13, 15, 17, 

S7 

Establish an internal interface working group 

14, 76, 77 



Define testability requirements as an integral part of top- 
level requirements specification 


Establish test working group at concept stage 


Develop generic wrapper to interface with NDS such that a 
different COTS product can be substituted 


Identify functional and performance requirements for 
capabilities to be provided by NDS and qualify the NDS 
against this early lifecycle 


Extend schedule to allow for unit testing 


Improve design 


Acquire additional equipment 


16 


18, 73 



S16 

Begin interface testing as early as practical, using 
simulations if necessary 

26, 73 

S17 

Identify maintainability issues early in lifecycle and 
prototype if possible 

28 

S18 

Perform additional V&V on maintenance items of concern 

28 

S19 

Identify reliability issues early in lifecycle and prototype if 
possible 

29 

S20 

Perform additional V&V on reliability items of concern 

29 

S21 

Initiate independent process reviews at outset 

36, 38, 39 

S22 

Review appropriate plans, e.g., Management Plan and 
Software Development Plan 

37 

S22A 

Perform tool suitability assessment 

37 

S23 

Perform process training 

39, 56, 60, 67 


Perform independent traceability verification 


Perform analysis to define needed development 
environment 


Acquire the right development environment 
Get training on development system from outset 


40 


42-44, 69 


42-44, 69 


45 
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Potential PACT Description h ; Mapping to ARRT Failure Modes 

Specialized 

swd pacts . 'A: 


128 Begin using DE as early as possible | 46 


S29 Make vendor deliver reliable product (warranty) 


S30 Develop a fallback plan 46, 47, 50 t 80 


S31 Perform an organizational review and implement 51 

recommendations 


S32 Utilize consultants and/or experienced managers to 52, 56 

augment - 


S33 Train managers 52, 54-56 


S34 Add more Technical Interchange Meetings and informal 53-55, 62, 76, 77 

staff meetings 


S35 Establish a User Working Group 53, 62 


S36 Management by Example (MBE); instill quality spirit 60, 61, 63 


S37 Conduct team building workshop 60, 61, 62, 63, 76, 77 


538 If early in SDLC, develop a staff training plan and implement 67, 53 

539 If late in SDLC, augment with experienced staff/mentors 67 
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6 ACRONYMS 


Acronym Expansion 

AHP 

Analytical Hierarchy Process 

ARRT 

Advanced Risk Reduction Tool 

CMM 

Capability Maturity Model 

CMMI 

Capability Maturity Model Integrated 

CO 

Contracting Officer 

COTR 

Contracting Officer’s Technical Representative 

CTO 

Contract Task Order 

DoD 

Department of Defense 

FM 

Failure Mode 

I&C 

Instrumentation and Control 

ISO 

International Organization for Standardization 

IT 

Information Technologies 

IV&V 

Independent Verification and Validation 

IWG 

Interface Working Group 

JPL 

Jet Propulsion Laboratory 

KPA 

Key Practice Area 

NASA 

National Aeronautics and Space Administration 

PACTS 

Preventive measures Analyses Controls Tests 

PM 

Program Manager 

POC 

Points of Contact 

QA 

Quality Assurance 

RE 

Risk Element 

RTCA 

Requirements and Technical Concepts for Aviation 

SAIC 

Science Applications International Corporation 

SBP 

Standard Best Practices 

SDP 

Software Development Plan 

SEI 

Software Engineering Institute 

SETA 

Science and Engineering Technical Assessments 

SLP 

System Level Procedure 

SOR 

Statement of Requirements 

SOW 

Statement of Work 

SP 

Special Practices 
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Acronym Expansion 

SPMN 

Software Program Managers Network 

VA 

Virginia 

WSRB 

Weapons Safety Review Board 

WV 

West Virginia 
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